1. Field of the Invention
The present invention relates generally to interior design, and in particular, to a method, apparatus, and article of manufacture for transferring the texture from one portion of an image to another portion of an image in a seamless manner in real-time.
2. Description of the Related Art
Given a single image or photograph of a room interior and a known three-dimensional (3D) model of the room, it is desirable to “clean” the room by concealing certain areas and regions in the photograph (e.g., using patches from other areas). For example, it may be desirable to copy a source patch of a rug to conceal a chair (the target patch), or to copy a patch of wall paper (source) to conceal a power-socket (target).
Using simple pixel copying to conceal an area/region has several problems that make such a copy technique unusable: (1) gradual lighting and illumination changes across the room cause a moved patch to appear either too dark or too light in its new position; and (2), perspective makes each patch unique and non-repeating in the image, so that even a repeating pattern in the world cannot be just moved inside the image.
With respect to the first problem relating to lighting and illumination, in prior art pixel manipulation applications (e.g., PHOTOSHOP™MSPAINT™etc.), a user may mark a rectangle in an image and simply copy and paste the rectangle to a new area. However, subtle lighting and illumination issues result in stark noticeable contrast in the borders of the resulting image. Further, if the outlines/borders are blurred, such blurring still fails to cure the lighting and illumination issues. In other words, the prior art systems fail to account for the internal illumination of a scene.
FIGS. 1A-1B illustrate the problems associated with illumination or surrounding brightness integration of the prior art. FIG. 1A illustrates object 102 that the user desires to remove from image 104. In other words, the user desires to clear/clean the image and remove object 102 from image 104. FIG. 1B illustrates the problems associated with prior art solutions. In particular, the user has identified a source patch 106 to be used to replace the object 102 and simply copies and pastes the source patch 106 over the object 102. As illustrated in FIG. 1B, the source patch 106 is merely copied and pasted resulting in target patch 108. The dashed line 110 illustrates the prior location of object 102. The result includes a target patch 108 that does not match the pattern/colors/illumination/brightness of the background image 104. To match the source and target, the user would have to find the exact match somewhere else in the image to copy and paste in the target. However, such an exact match may not exist, and/or is extremely difficult to find and select.
Even if prior art systems are capable of overcoming the lighting and illumination issues (e.g., using “seamless cloning” techniques), the second problem (i.e., perspective) still exists. For example, scenes are often three-dimensional and contain perspective. In a 3D picture, if a user moves a patch from one area to another, the perspective will not align properly. For example, if a room includes wallpaper with a pattern, checkered tile flooring, etc., when a patch is applied from one area to another area, the patterns/lines will not converge in the right direction. Further, a scaling issue will likely exist.
FIGS. 2A-2B illustrate the problem associated with perspective. FIG. 2A illustrates an object 202 that the user desires to remove from a 2D image 204 of a world plane. FIG. 2B illustrates the selection of a source patch 206 that will be copied and pasted over the selected target patch 208. As illustrated, the perspective of source patch 206 does not match the perspective of the target patch 208. Due to the difference in perspective, the pattern from the source patch 206 will not match the pattern in image 204 if pasted onto target patch 208. In other words, the checkered pattern boxes within target patch 208 would have a different shape and different angle from what would be expected at that location in image 204.
In view of the above, mathematically, there is no patch within one image that would be the right fit for another patch/area in an image that the user desires to conceal (except, perhaps, in some very specific degenerate cases). Further, prior art seamless cloning techniques introduce diffused colors when patches are copied across high contrast areas. Accordingly, what is needed is the capability to conceal an area of an image using a patch from within the image, in a seamless manner, in real-time, while accounting for lighting/illumination changes as well as perspective.